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DETAILED ACTION 

1 . This office action is filed in response to the application filed on 2/1 3/2004, IDS 
filed on 10/05/2005, and IDS filed on 2/13/2004. 

2. Claims 1-20 are pending. Claims 1, 8, 11, 13, 14, 17, 19, and 20 are independent 
claims. 

Specification 

3. The abstract of the disclosure is objected to because the abstract is too short 
(less than 50 words). See comments below or MPEP § 608.01(b). 

Applicant is reminded of the proper language and format for an abstract of the 
disclosure. The abstract should be in narrative form and generally limited to a single 
paragraph on a separate sheet within the range of 50 to 150 words. It is important that 
the abstract not exceed 150 words in length since the space provided for the abstract 
on the computer tape used by the printer is limited. The form and legal phraseology 
often used in patent claims, such as "means" and "said," should be avoided. The 
abstract should describe the disclosure sufficiently to assist readers in deciding whether 
there is a need for consulting the full patent text for details. The language should be 
clear and concise and should not repeat information given in the title. It should avoid 
using phrases which can be implied, such as, "The disclosure concerns," "The 
disclosure defined by this invention," "The disclosure describes," etc. 

4. The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. 

The following title is suggested: 
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'Organization of Annotated Clipping Views'. 
Correction is required. 

Claim Rejections -35 USC §102 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

5. Claims 1, 2, 5, 8, 11, 14, and 17 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Schilit et al (US Patent: US 6,279,014 B1, issued: Aug. 21, 2001, filed: 
Sep. 15, 1997). 

With regards to claim 1, Schilit et al teaches a system comprising: 

a) An input for receiving said content and said annotations (paragraph 0016: Whereas, 
memory 16, and 1 8 store/input the content and annotations respectively). 

b) A processor creating a renderable image having said clips, wherein at least one of 
said clips is a combination of two or more annotations: The processor renders an 
image having clips as shown in Fig. 5 and described in column 6, lines 5-23. Each 
clip can contain more than one user drawn digital ink annotations that are combined 
together based on time and space as described in paragraph column 5, lines 10-32. 

c) An output for outputting said renderable image (Fig. 4 is an output of the renderable 
image, as described in column 6, lines 5-23). 

With regards to claim 2, Schilit et al teaches a system according to claim 1, 
wherein said at least one of said clips includes additional content: whereas each clip 
comprises a context portion (Schilit et al, column 6, lines 45-50), and addition content 
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information, such as attributes concerning the type (column 7, lines 23-26), identified 
content (column 7, lines 27-30), or time/date data (column 7, lines 35-39). 

With regards to claim 5, Schilit et al teaches a system according to claim 1 , 
further comprising a storage storing said annotation (column 3, Fig. 1 , reference number 
18: whereas, annotations are stored on storage device 18) and an image of said content 
(column 3, Fig. 1, reference number 16: whereas, an image of said content is stored on 
storage device 16). Furthermore, "memories 16 and 18 can be either distinct portions of 
a single memory" (column 4, Fig. 1, lines 14-15). 

With regards to claim 8, Schilit et al teaches a method of displaying clips 
comprising the steps of: 

a) Receiving at least two sets of an annotation and related content: whereas, as shown 
in Fig. 3, each set of user drawn digital ink annotations (such as the first set 36, and a 
second set 38) and related content (context data and attributes in Step S260, Fig. 6) are 
received and processed by a process function block S280 in Fig. 6. 

b) Combining said at least two sets: (column 6, lines 5-23: whereas, the sets are 
combined into a single ordered list prior to display/output). 

c) Displaying a combination of said at least two sets (column 6, lines 5-23, Fig. 4: 
whereas, the combination of set 36, and set 38 are displayed into a single output 
display as shown in Fig. 4). 

With regards to claim 1 1 , Schilit et al teaches a method of displaying clips 
comprising the steps of: 
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a) Receiving at least two sets of an annotation and related content, as similarly 
described in claim 8, and is rejected under the same rationale. 

b) Filtering said at least two sets: whereas, each annotation set can be filtered into 
different list views such as by attribute, type of adjacent material, or content of adjacent 
material (column 6, lines 5-23). 

c) Combining the output of said filtering step (column 6, lines 5-23: whereas, each list 
view contains output of the sets based upon the method of filtering chosen). 

d) Displaying a combination of said filtering step (column 6, lines 5-23: whereas, "the 
system visually presents the annotations in context using different list views"). 

With regards to claim 14, Schilit et al teaches a computer-readable medium 
having a program stored thereon, said program for displaying clips and comprising the 
steps of: 

a) Receiving at least two sets of an annotation and related content: as similarly 
described in claim 8, and is rejected under the same rationale. 

b) Combining said at least two sets, as similarly described in claim 8, and is rejected 
under the same rationale. 

c) Displaying a combination of said at least two sets, as similarly described in claim 8, 
and is rejected under the same rationale. 

With regards to claim 1 7, Schilit et al teaches a computer-readable medium 
having a program stored thereon, said program for displaying clips and comprising the 
steps of: 
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a) Receiving at least two sets of an annotation and related content, as similarly 
described in claim 1 1 , and is rejected under the same rationale. 

b) Filtering said at least two sets, as similarly described in claim 1 1 , and is rejected 
under the same rationale. 

c) Combining the output of said filtering step, as similarly described in claim 1 1 , and is 
rejected under the same rationale. 

d) Displaying a combination of said filtering step, as similarly described in claim 1 1 , and 
is rejected under the same rationale. 

Claim Rejections -35 USC §103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over 

Schilit etal (US Patent: US 6,27.9,014 B1, issued: Aug. 21, 2001, filed: Sep. 15, 1997), 

in further view of Ma et al (US Patent: 6,909,805 B2, published: Jun. 21, 2005, filed: 

Jan. 31, 2001). 

With regards to claim 3, which is dependent on claim 1 , Schilit et al teaches the 
use of bounding boxes for each annotation (Fig. 5, reference number 34, column 5, 
lines 8-9: whereas, a clip comprising a bounding box, is used to select a context portion 
based on an annotation). However, Schilit et al.does not teach a clip is a combination of 
two or more bounding boxes, said bounding boxes surrounding said annotations. 
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Ma et al teaches a clip is a combination of two or more bounding boxes, said 
bounding boxes surrounding said annotations: whereas, a clip of annotation images 
extracted from an original scanned document is extracted in Fig 8. Furthermore, clip is a 
combination of bounding boxes surrounding annotations as shown in Fig. 7. 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified Schilit et al's invention such that each segmented clip, 
includes a combination of two or more bounding boxes surrounding said annotations as 
taught by Ma et al. The combination of Schilit et al and Ma et al to have allowed Schilit 
et al's clips that each contain a bounding box, to further include the ability to contain a 
combination of two or more bounding boxes, for optimal manipulation of annotated data 
by reducing the number of boxes or image segments handled. 

With regards to claim 4, which is dependent on claim 1, Schilit et al and Ma et al 
teaches the clips are a combination of two or more bounding boxes, where said 
bounding boxes surround said annotations, as explained in claim 3 above. Furthermore, 
each bounding box defines a region/area of the document of interest, as shown in Fig. 7 
of Ma et al. Since each bounding box defines a region of interest, then claim 4, for a 
system performing the same method as the method in claim 3, is rejected under the 
same rationale. 

7. Claims 6, 7, 10, 12, 13, 16, 18, 19, and 20 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Schilit et al (US Patent: US 6,279,014 B1, issued: Aug. 21, 
2001, filed: Sep. 15, 1997), in further view of Amitay et al (US Application: US 
2004/0216032 A1, published: Oct. 28, 2004, filed: Apr. 28, 2003). 
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With regards to claim 6, which is dependent on claim 1 , for a system performing 
a method similar to claim 13, is rejected under the same rationale. 

With regards to claim 7, which is dependent on claim 1 , Schilit et al teaches a 
system for storing annotations in electronic documents (column 4, lines 39-40) and 
content (Fig. 1 , reference number 16, column 3, lines 55-57) and content (Fig. 1, 
reference number 18, column 3, lines 62-63). However, Schilit et al does not teach 
storing active content. 

Amitay et al teaches a storage storing said annotation and active content: 
whereas, annotations are stored along with an "image of at least a portion of the 
computer screen viewed by the user" (paragraph 0016). Furthermore, the content 
includes active content such as video (paragraph 0039: whereas, annotated electronic 
documents, include video or electronic whiteboards). 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified Schilit et al's system for storing annotations and content, to 
also include the storing of active content through the use of snap shot images as taught 
by Amitay et al. The combination of Schilit et al and Amity et al would have allowed 
Schilit et al's system to have organized and stored annotations for not only static 
content, but active content as well such that the context of the active content is 
preserved upon retrieval. 

With regards to claim 10, which depends on claim 8, Schilit et al teaches a 
method for combining said at least two sets such that the combinations are represented 
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in list form, in claim 8, and is rejected under the same rationale. However, Schilit et al 
does not expressly teach storing the combination of said at least two sets are stored. 

Amitay et al teaches combining annotations in a sorted list, for which the list is 
sorted in memory (paragraph 0063: whereas, a combination/set of annotations are 
sorted/manipulated in memory and thus, stored). 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified the Schilit et al's annotation combinations to be stored in 
memory as taught by Amity et al. The combination of Amitay et al and Schilit et al would 
have allowed Amitay et al's system to have stored a combination for later retrieval, thus 
reducing processing time. 

With regards to claim 12, which depends on claim 11, for a method performing a 
method similar to claim 10, is rejected under the same rationale. 

With regards to claim 13, Schilit et al teaches a method of storing and accessing 
clips comprising the steps of: 

a) Receiving data regarding a first annotation and data regarding a second annotation: 
as similarly described in claim 8, and is rejected under the same rationale. 

b) Storing at least one of context information with said first and second annotation data 
or a link to context information with said first and second annotation data in a storage: 
whereas, for each annotation, context information is stored as described in column 9, 
lines 1-8. As shown in the input, there are at least two sets of user drawn annotations 
(36, and 38), and using the procedure in Fig. 6, both first and second annotation data 
are stored. 
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However, Schilit et al does not teach storing associations regarding one or more 
applications from which said first annotation and second annotation originate and 
wherein selection of one of said annotations accesses said one or more applications to 
display said annotations. 
Amitay et al teaches a method for: 

a) Storing associations regarding one or more applications from which said first 
annotation and second annotation originate (paragraph 0060: whereas, each annotation 
is stored such that there are additional fields of context data, including the application 
type of origin). 

b) Wherein selection of one of said annotations accesses said one or more applications 
to display said annotations (paragraph 0071 : whereas, a workstation selects one or 
more annotations for display by following a link to the document using said one or more 
applications). 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified Schilit et al's context information for annotations, to further 
include the application type to indicate where the annotations have originated, and used 
the application type data to open the appropriate application when a selection for one or 
more annotations are made, as taught by Amitay et al. The combination of Schilit et al 
and Amitay et al would have allowed Schilit et al's system to have been able to 
"organize their annotations coherently and to review the annotations easily, even when 
the annotations extend over multiple, different electronic documents, multiple different 
screens" (Amitay et al, paragraph 0006). 
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With regards to claim 16, which depends on claim 14, for a computer readable 
medium having a program stored thereon, for performing a method similar to the 
method of claim 10, is rejected under the same rationale. 

With regards to claim 18, which depends on claim 17, for a computer readable 
medium having a program stored thereon, for performing a method similar to the 
method of claim 10, is rejected under the same rationale. 

With regards to claim 19, for a computer readable medium having a program 
stored thereon, for performing a method similar to the method in claim 13, is rejected 
under the same rationale. 

With regards to claim 20, Schilit et al teaches a computer-readable medium 
having stored thereon: 

a) Storing information relating to context information (column 9, lines 5-9), for which the 
context information includes at least one attribute of context information, and additional 
attributes of context information as described in column 7, lines 12-44. 

However, Schilit et al does not teach a data structure stored thereon, said data 
structure comprising a first field storing information relating to context information, a 
second field storing information relating to an annotation that adds information to said 
context information, and a third field storing a link to a document related to said context 
information, such that activation of said link opens said document. 

Amitay et al teaches a computer readable medium having a data structure stored 
thereon, said data structure comprising: 
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a) A first field storing information relating to context information (paragraph 0042: 
whereas, a first field stores time related context information for a particular annotation). 

b) A second field storing information relating to an annotation that adds information to 
said context information (paragraph 0048: whereas, a second field stores the identity, of 
the user creating the annotation) 

c) A third field storing a link to a document related to said context information, such that 
activation of said link opens said document whereas, a third field stores the file name 
and application type of the document (paragraph 0043). Furthermore, the link opens 
the document (paragraph 0071: whereas, a workstation selects one or more 
annotations for display by following a link to the document using said one or more 
applications). 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified Schilit et al's storage of context information, to also store 
information in data structure comprising a set of fields, such that the fields contain 
additional context information, including a link used to open the document relating to the 
context information as taught by Amitay et al. The combination of Schilit et al and 
Amitay et al would have allowed Schilit et al's system to have been able to "store 
context information .... in a structured, hierarchical format" (Amitay et al, paragraph 
0050). 

8. Claims 9 and 15 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Schilit etal (US Patent: US 6,279,014 B1, issued: Aug. 21, 2001, filed: Sep. 15, 1997), 
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in further view of Hugh et al (US Application: US 2002/0089551 A1 , published: Jul. 1 1 , 
2002, filed: Jul. 13, 2001). 

With regards to claim 9, which depends on claim 8, Schilit et al teaches filtering at least 
two sets, as explained in claim 1 1 above, and is rejected under the same rationale. 
Furthermore, Schilit et al teaches a combination of two sets, as described in the 
rejection for claim 8 above, and is rejected under the same rationale. However Schilit et 
al does not teach filtering said combination of said at least two sets. 
Hugh et al teaches a method for filtering said combination of annotation data: whereas, 
filtering can be performed on a combination/set of annotation data: whereas, filtering is 
performed using a result set of annotation/thought combinations (paragraph 0361). For 
clarification, the "thoughts" being filtered or organized comprise annotation data as 
described in paragraph 0563. 

It would have been obvious to one of the ordinary skill in the art at the time of the 
invention to have modified Schilit et al's methods for filter and combining annotations, to 
also include the ability to perform the method of combing prior to filtering as taught by 
Hugh et al. The combination of Schilit et al and Hugh et al would have allowed Schilit et 
al's system to have provided a "filtered (result set which) can be tailored to suit the 
user's display preferences" (paragraph 0016). 

With regards to claim 15, which is dependent on claim 14, for a computer-readable 
medium having a program stored thereon, for which the said program performs a 
method similar to the method of claim 9, is rejected under the same rationale. 

Conclusion 



Application/Control Number: 10/777,046 Page 14 

Art Unit: 2178 

9. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

• Bays et al (US Patent: US 6,519,603 B1, published: Feb. 11, 2003, filed: 
Oct. 28, 1999): This reference teaches methods for organizing and 
retrieving annotations. 

• Golovchinsky et al (US Application: US 2004/0078757 A1 , published: Apr. 
22, 2004, filed: Aug. 31 , 2001): This reference teaches methods for 
processing annotation data, digital ink, bounding boxes, and annotation 
linking. 

• Price et al (US Patent: 6,766,494 B1 , published: Jul. 20, 2004, filed: Jun. 
1 5, 1988): This reference teaches linking annotations and storage/use of 
contextual data. 

• Lerner et al (US Patent: US 6,859,909 B1 , published: Feb. 22, 2005, filed: 
Mar. 7, 2000): This patent teaches annotating web documents, selecting 
portions of a web document, and organizing web annotated documents. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Wilson Tsui whose telephone number is (571)272-7596. 
The examiner can normally be reached on Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Stephen Hong can be reached on (571) 272-4124. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). I 



Wilson Tsui 
Patent Examiner 
Art Unit: 2178 
March 22, 2006 
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